Neural network for bulk sorting

ABSTRACT

A bulk sorting system for sorting objects in bulk is provided. The bulk sorting system includes: at least one radiation source arranged to radiate the objects, at least one optical sensor arranged to capture reflected radiation of the objects and acquire the reflected radiation as multi- or hyperspectral data; a processing circuit configured to analyze the reflected radiation of the objects by inputting the multi- or hyperspectral data into a convolutional neural network (CNN) with at least two convolutional layers in order to either detect and classify the objects in the multi- or hyperspectral data and/or semantically segment the multi- or hyperspectral data; and a mechanical sorter configured to sort the objects according to their classification and/or segmentation using the analysis of the processing circuit such that different overlapping and/or stacked objects are separated or treated as a single group of objects.

TECHNICAL FIELD

The present invention relates to the field of sorting. Moreparticularly, the present invention relates to bulk sorting aided by aconvolutional neural network (CNN).

BACKGROUND

Sorting is a topical field of research with implications for e.g.recycling, mining or food processing. For a recycling implementation,sorting techniques are used to sort a mixture of garbage into thecorrect recycling bin. As technology evolves, this sorting may be donemore accurately and faster than before.

There exists techniques such as those shown in US 2018/243800 AA, forusing a machine learning system for sorting a stream of single objects.The machine learning system allows for an accurate identification of theobjects being sorted. However, such techniques are slow as they may onlyprocess a single stream of objects at a time. Other examples of materialcharacterization and segmentation techniques may be found in US2019/130560 A1 and Matthieu Grard et al: “Object segmentation in depthmaps with one user click and a synthetically trained fully convolutionalnetwork”, 2018.

Current sorters separate individual particles. They require careful feedpreparation so that individual particles may be detected and measured,and ejection is usually achieved by blasts of compressed air. Therefore,current sorters have very low capacity (up to 300 tonnes per hour forlarger particles and much less for smaller particles), making themnon-viable for higher tonnage pre-concentration or so-called bulksorting. The sorting speed and throughput to be achieved in bulk sortingis directly related to the size of the objects to be sorted. In general,sorting speed and throughput greatly varies for larger and smallerarticles and is much dependent on the particular application which mayrange from food, paper, wood, plastic and mineral sorting applications.

To make sorting viable for all pre-concentrations, it should be appliedto bulk quantities of objects, such as on a loaded truck tray or a fullyloaded conveyor belt.

There is thus a need for improvements within this context.

SUMMARY

Thusly, the present invention strives to solve at least some of theabove problems and to eliminate or at least mitigate some of thedrawbacks of prior-art systems. This object has now been achieved inaccordance with the invention by the novel technique set forth in theappended independent claims; preferred embodiments being defined in therelated dependent claims.

A further object of the present invention is to provide a sorting systemcapable of sorting objects in bulk. According to a first aspect, theabove and other objects of the invention are achieved, in full or inpart, by a bulk sorting system for sorting objects in bulk. The systemcomprises: at least one radiation source arranged to radiate theobjects, at least one optical sensor arranged to capture reflectedradiation of the objects and acquire the reflected radiation as multi-or hyperspectral data; a processing circuit configured to analyze thereflected radiation of the objects by inputting the multi- orhyperspectral data into a convolutional neural network (CNN) with atleast two convolutional layers in order to either detect and classifythe objects in the multi- or hyperspectral data and/or semanticallysegment the multi- or hyperspectral data; and a mechanical sorterconfigured to sort the objects using the analysis of the processingcircuit.

This is beneficial in that previous problems such as objects laying ontop of each other, thus obstructing sensor data, may be identified andprocessed accordingly. The CNN with at least two convolutional layersmay advantageously be trained to handle this type of multi- orhyperspectral data in an improved way compared to known sorting systemsfor bulk sorting. The detection and classification and/or semanticsegmentation with the CNN allows performing object processing withoverlapping objects, which leads to higher possible sorter throughputper hour compared to traditional processing methods.

In one embodiment, the at least one optical sensor comprises anear-infrared scanner arranged to scan the objects, wherein the multi-or hyperspectral data comprises the scan data.

The NIR scanner is beneficial in that a lot of distinctive informationmay be extracted from measured NIR absorption of the objects.

In one embodiment, the at least one optical sensor comprises an imagesensor arranged to capture image data of the objects, wherein the multi-or hyperspectral data comprises the image data.

The image sensor is beneficial in that many image processing algorithmsexists for image data, such as object recognition or image segmentation.Moreover, a CNN is typically well suited for analysis of image data.

In one embodiment, the at least one optical sensor comprises ahyperspectral camera arranged to scan the objects, wherein the multi- orhyperspectral data comprises the scan data.

The hyperspectral camera is beneficial in that it is an efficient way togather a lot of data to be used by the CNN.

In one embodiment, the at least one optical sensor comprises a lasertriangulator arranged to measure 3D-data of the objects, wherein themulti- or hyperspectral data comprises the 3D-data, wherein themeasurement of 3D-data may comprise laser height intensity scanning.

The laser triangulator is beneficial in that it allows a precise3D-measurement, which is beneficial for the analysis of the CNN.

In one embodiment, the system further comprises an electromagneticdetector arranged to measure electromagnetic properties of the objects,wherein the processing circuit is further configured to analyze theelectromagnetic properties by inputting the measured electromagneticproperties into the CNN in order to either detect and classify theobjects in the multi- or hyperspectral data and/or semantically segmentthe multi- or hyperspectral data.

The electromagnetic detector is beneficial in that classification ofcertain types of objects, such as metals and isolators, is much moreprecise using electromagnetic properties.

In one embodiment, the at least one optical sensor comprises a laserscanner with a rotating polygon mirror arranged to measure laser scatterand/or anti-scatter properties of the objects; wherein the multi- orhyperspectral data comprises the laser scatter and/or anti-scatterproperties.

The laser scanner is beneficial in that a lot of distinctive informationmay be extracted from measured laser scatter and/or anti-scatterproperties of the objects. The rotating polygon mirror allows severalobjects and scattering characteristics in different directions to bemeasured without any targeting.

In one embodiment, the at least one optical sensor comprises a pulsedLED emitter arranged to measure light anti-scatter properties of theobjects; wherein the multi- or hyperspectral data comprises the lightanti-scatter properties.

The LED emitter is beneficial in that they require little maintenanceand calibration and generate relatively little heat. A lot ofdistinctive information may be extracted from measured lightanti-scatter properties of the objects.

In one embodiment, the at least one optical sensor comprises an X-raycamera arranged to measure X-ray transmission of the objects; whereinthe multi- or hyperspectral data comprises the X-ray transmission of theobjects, respectively.

The X-ray camera is beneficial in that different properties may bemeasured at once. The permittivity of the objects to X-ray radiation mayindicate their atomic density and/or their thickness. The result ofX-ray fluorescence is information about presence of elements and theirconcentration.

In one embodiment, the system further comprises conveying the objectsalong detection range(s) of the at least one optical sensor to themechanical sorter using a conveyor belt.

The conveyor belt is beneficial in that it allows stable, fast andpredictable motion of the objects. An alternative to the conveyor beltis a chute.

In one embodiment, the mechanical sorter is further configured toseparate the objects into at least two streams and/or to eject unwantedobjects from the bulk.

The separation of the objects is beneficial in that several sets ofobjects may be wanted from the bulk. The ejection of unwanted objects isbeneficial in that there is frequently unwanted objects in the bulk thatinterfere with future processes.

In one embodiment, the mechanical sorter comprises at least one valvearranged to eject at least one air stream that pushes the objects to adesired position.

The at least one valve is beneficial in that it is an energy efficientand precise way to push objects. The valves are preferably solenoidvalves, as they have relatively short switching times and a relativelyhigh throughput performance.

In one embodiment, the mechanical sorter comprises at least onemechanical kicker arranged to kick the objects to a desired position.

The at least one mechanical kicker is beneficial in that it is an energyefficient and powerful way to push objects. The kickers are preferablypneumatic.

In one embodiment, the mechanical sorter is arranged to sort the objectsas they travel along a conveyor belt, as they travel along a chute or asthey freefall.

The sorting while travelling is beneficial in that it is efficient interms of speed. The sorter may be arranged near the conveyor belt andarranged to move the objects in a direction perpendicular to thedirection of motion of the conveyor belt. The chute or freefall may bearranged at an end of a conveyor belt or along an edge of a conveyorbelt. It is preferential that the mechanical sorter is capable ofreacting quickly as the velocity of the objects are relatively hard topredict and control in this embodiment. In one embodiment, themechanical sorter has an activation precision of less than 1 ms forsmall particles and less than 10 ms for bigger particles, such asbottles.

In one embodiment, the mechanical sorter is arranged to sort the objectssuch that different overlapping and/or stacked objects, as analyzed bythe processing circuit, are separated.

The separation of overlapping and/or stacked objects is beneficial inthat different objects may be treated differently despite beingoverlapping and/or stacked, which increases accuracy and yield of thesorting.

In one embodiment, the mechanical sorter is arranged to sort the objectssuch that different overlapping and/or stacked objects, as analyzed bythe processing circuit, are treated as a single group of objects.

The grouping of overlapping and/or stacked objects is beneficial in thatit may be difficult to separate them, especially with certain mechanicalsorters.

In one embodiment, overlapping and/or stacked objects are treated as asingle group of objects, wherein for a group of objects comprising atleast a first object type and a second object type, the mechanicalsorter is configured to sort the group of objects as either the firstobject type or the second object type based on a preference in the bulksorting system.

The sorting of a group as a single object type is beneficial in thatdepending on the embodiment, it may be more valuable to ensure that noneof the wanted material is ejected or that no unwanted material is kept.In the first case, the group would be sorted as the wanted object typeand in the second case, the group would be sorted as the unwanted objecttype.

In one embodiment, the mechanical sorter is arranged to target thecenter of gravity or boundaries of the objects, as analyzed by theprocessing circuit.

The targeted mechanical sorting is beneficial in that it allows themechanical sorter to be more efficient in that e.g. fewer valves orkickers are used and the objects are moved without rotating, whichincreases accuracy and lowers energy consumption. For example, a thirdof mechanical sorters may be used for the same result if they aretargeted better.

Object sorting, depending on the size of the objects to be sorted, at aspeed of between 0.4-20 m/s and a throughput of 0.5-30 tons/hr at anaccuracy level of higher than 80%, preferably higher than 90% with asingle stage throughput and higher than 95%, preferably higher than 99%with a cascade of sorting systems that may be achieved with the sortingsystem and method of present application.

In one embodiment, the CNN comprises at least two pooling layers.

The pooling layers are beneficial in that they reduce the size of theneural network by down-sampling the data, which makes the CNN moreefficient. The CNN may have any number of pooling layers, including 3,5, 10, 20, 100, 250, etc.

In one embodiment, the system further comprises post-processing theclassified and/or segmented multi- or hyperspectral data to configurethe mechanical sorter before sorting occurs.

The post-processing is beneficial in that it allows converting of theoutput into something the mechanical sorter may more easily interpret.This may comprise configuration instructions for the mechanical sorterto be created before sorting occurs of the objects corresponding to theclassified and/or segmented multi- or hyperspectral data.

In one embodiment, the processing circuit is further configured to inputat least a part of the multi- or hyperspectral data into a patternrecognition algorithm; and wherein the results of both the CNN and thepattern recognition algorithm are used by the mechanical sorter to sortthe objects.

The pattern recognition is beneficial in that it is well established andmay enhance the result of the CNN. It also does not require any trainingand may be completed relatively quickly, which may be used if the CNN isunavailable or too slow. Such a hybrid computation has unexpectedsynergistic benefits as different information may be obtained by thedifferent analysis methods.

According to a second aspect, the above and other objects of theinvention are achieved, in full or in part, by a method for sortingobjects in bulk. The method comprises steps of: radiating the objectsusing at least one radiation source; capturing reflected radiation ofthe objects using at least one optical sensor; acquiring the reflectedradiation as multi- or hyperspectral data; analyzing the reflectedradiation of the objects by inputting the multi- or hyperspectral datainto a convolutional neural network (CNN) with at least twoconvolutional layers in order to either detect and classify the objectsin the multi- or hyperspectral data and/or semantically segment themulti- or hyperspectral data; and sorting, by a mechanical sorter, theobjects using the results of the analysis step.

This is beneficial in that previous problems such as objects laying ontop of each other, thus obstructing sensor data, may be identified andprocessed accordingly. The detection and classification and/or semanticsegmentation with the CNN allows performing object processing withoverlapping objects, which leads to higher possible sorter throughputper hour compared to traditional processing methods. The sorterthroughput per hour may be increased by at least 50% compared totraditional processing methods. Object sorting, depending on the size ofthe objects to be sorted, at a speed of between 0.4-20 m/s and athroughput of 0.5-30 tons/hr may be achieved.

In one embodiment, the method further comprises a step ofpost-processing the classified and/or segmented multi- or hyperspectraldata to configure the mechanical sorter before the sorting step.

The post-processing step is beneficial in that it allows converting ofthe output into something the mechanical sorter may more easilyinterpret. This may comprise configuration instructions for themechanical sorter to be created before the sorting step of the objectscorresponding to the classified and/or segmented multi- or hyperspectraldata.

Other objectives, features and advantages of the present invention willappear from the following detailed disclosure, from the attached claims,as well as from the drawings. It is noted that the invention relates toall possible combinations of features.

It should be emphasized that the term “comprises/comprising” when usedin this specification is taken to specify the presence of statedfeatures, integers, steps, or components, but does not preclude thepresence or addition of one or more other features, integers, steps,components, or groups thereof. All terms used in the claims are to beinterpreted according to their ordinary meaning in the technical field,unless explicitly defined otherwise herein. All references to “a/an/the[element, device, component, means, step, etc.]” are to be interpretedopenly as referring to at least one instance of the element, device,component, means, step, etc., unless explicitly stated otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

By way of example, embodiments of the present invention will now bedescribed with reference to the accompanying drawings, in which:

FIG. 1 shows a bulk sorting system according to an embodiment;

FIG. 2 shows a bulk sorting system according to an embodiment;

FIG. 3 a shows a result of object detection according to an embodiment;

FIG. 3 b shows a result of semantic segmentation according to anembodiment;

FIG. 4 shows a mechanical sorter according to an embodiment;

FIG. 5 a shows a bulk sorting system comprising a near-infrared scannerand an image sensor according to an embodiment;

FIG. 5 b shows a bulk sorting system comprising a laser triangulator andan electromagnetic detector according to an embodiment;

FIG. 6 a shows a bulk sorting system comprising a laser scanner, apulsed LED emitter and an X-ray camera according to an embodiment;

FIG. 6 b shows a bulk sorting system comprising a hyperspectral cameraaccording to an embodiment;

FIG. 7 shows a flowchart for operations of the processing circuitaccording to an embodiment;

FIG. 8 shows a flowchart for hybrid operations of the processing circuitaccording to an embodiment;

FIG. 9 shows a flowchart for hybrid operations of the processing circuitusing two computing units according to an embodiment; and

FIG. 10 shows a flowchart for a method for sorting objects in bulkaccording to an embodiment.

DETAILED DESCRIPTION

Embodiments of the invention will now be described with reference to theaccompanying drawings. The invention may, however, be embodied in manydifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete and will fully conveythe scope of the invention to those skilled in the art. The terminologyused in the detailed description of the particular embodimentsillustrated in the accompanying drawings is not intended to be limitingof the invention. In the drawings, like numbers refer to like elements.

Starting in FIGS. 1 and 2 , a bulk sorting system is shown, wherein FIG.2 shows a more detailed example of the same bulk sorting system as inFIG. 1 . The system is configured to sort objects 1 in bulk. Objects 1in bulk may comprise a relatively large number of objects 1 that arerandomly spatially separated. Objects 1 in bulk are traditionally not asingle stream, but instead an unsorted set with objects 1 that mayoverlap. Bulk sorting traditionally aims to separate a specific set orsets of objects 1 from other objects 1, i.e. that some objects arewanted, and other objects are unwanted. One example is to separateplastic from other types of garbage, so that the plastic may berecycled.

The system comprises at least one radiation source 10 arranged toradiate the objects 1. The radiation source 10 may be an LED, a lightbulb, a laser, an X-ray emitter and/or any other suitable radiationsource. The radiation source 10 may be arranged as different units, asambient illumination and/or to radiate in different directions.

The system further comprises at least one optical sensor 12 arranged tocapture reflected radiation 22 of the objects 1 and acquire thereflected radiation 22 as multi- or hyperspectral data 24. Themultispectral data 24 may be a multi-dimensional image where each pixelor equivalent (such as super pixel or grouping of pixels) comprises afew bands (order of magnitude 1-10) with narrow or wide wide spectrums(order of magnitude 100-1000 nm). The hyperspectral data 24 may be amulti-dimensional image where each pixel or equivalent (such as superpixel or grouping of pixels) comprises many bands (order of magnitude4-500) with narrow spectrums (order of magnitude 1-50 nm).

The system further comprises a processing circuit 16 configured toanalyze the reflected radiation 22 of the objects 1 by inputting themulti- or hyperspectral data 24 into a convolutional neural network(CNN) 18 with at least two convolutional layers in order to eitherdetect and classify the objects 1 in the multi- or hyperspectral data 24and/or semantically segment the multi- or hyperspectral data 24. Morethan two convolutional layers may be used, such as 3, 5, 10, 20, 100,250, etc.

The processing circuit 16 is configured to carry out operations andfunctions of the bulk sorting system. Operations may be main proceduresof the bulk sorting system, whereas the functions may be forming part ofan operation. Hence, each function may be a sub-procedure of anoperation.

The processing circuit 16 may include a processor, such as a centralprocessing unit (CPU), microcontroller, microprocessor,field-programmable gate array (FPGA), graphic card, or special hardwarefor CNNs. The processing circuit 16 is configured to execute programcode stored in a memory, in order to carry out the operations andfunctions of the bulk sorting system.

Operations and functions of the bulk sorting system may be embodied inthe form of executable logic routines (e.g., lines of code, softwareprograms, etc.) that are stored on a non-transitory computer readablemedium (e.g., the memory) of the bulk sorting system and are executed bythe processing circuit 16 (e.g., using the processor). Furthermore, theoperations and functions of the bulk sorting system may be a stand-alonesoftware application or form a part of a software application thatcarries out additional tasks related to the bulk sorting system. Thedescribed operations and functions may be considered a method that thecorresponding device is configured to carry out. Also, while thedescribed operations and functions may be implemented in software, suchfunctionality may as well be carried out via dedicated hardware orfirmware, or some combination of hardware, firmware and/or software.

The CNN 18 is a neural network with at least two convolutional layers.The neural network 18 has an input of the multi- or hyperspectral data24 and an output of the multi- or hyperspectral data 24 with objectsdetected and classified and/or semantically segmented. An example of theoutput may be seen in FIGS. 3 a -b.

The CNN 18 may further comprise at least two pooling layers. The poolinglayers reduces the size of the neural network by down-sampling the data.This makes the CNN 18 more efficient.

Detecting and classifying objects in the multi- or hyperspectral data 24comprises using the CNN 18 (optionally in conjunction withpre-processing, see further below) to classify different parts of themulti- or hyperspectral data 24 as different objects 1. The objects 1may further be analyzed and classified into different sets, such asbased on the material that the objects 1 are most likely made of. Thedifferent sets may further be identified as wanted or unwanted, in orderto enable bulk sorting such as ejection of unwanted objects 1.

Semantically segmenting objects in the multi- or hyperspectral data 24comprises using the CNN to classify each pixel of the multi- orhyperspectral data 24. The classification may comprise separation intodifferent sets, such as based on the material that the pixel is mostlikely made of. The different sets may further be identified as wantedor unwanted (either by the CNN 18 or in conjunction with pre-processing,see further below), in order to enable bulk sorting such as ejection ofunwanted objects 1.

FIG. 3 a shows multi- or hyperspectral data with detected and classifiedobjects. The detection is shown as boxes around the objects and theclassification is shown as patterns of the objects. The detected objectsare classified into two sets, wanted and unwanted. The unwanted set isshown as a lined pattern with a dashed box and the wanted set is shownas a dotted pattern with a solid box.

FIG. 3 b shows multi- or hyperspectral data with semantically segmentedobjects. Each pixel or equivalent is classified into a set, for examplebased on the material of the objects according to the CNN. Backgroundpixels or equivalents are classified as such and disregarded by themechanical sorter. Each object is segmented and given a differentpattern. The patterns may correspond to a classification. Some objectsare overlapping, and the CNN has managed to differentiate them asdifferent objects and given them different patterns. The mechanicalsorter may be instructed to process these as different objects or asingle group of objects, and they may be classified accordingly.

The system further comprises a mechanical sorter configured to sort theobjects using the analysis of the processing circuit. The mechanicalsorter may be at least one valve, mechanical kicker, robot arm or anyother suitable device capable of mechanically moving objects.

The at least one valve is arranged to eject at least one air stream thatpushes the objects to a desired position. The desired position may be adifferent part of the bulk, a chute or container arranged in thedirection of the at least one valve or simply of the bulk sortingsystem. The at least one valve is preferably a solenoid valve, as theyhave relatively short switching times and a relatively high throughputperformance.

The valves may be arranged in an array with a distance between valves inan order of magnitude from mm to cm. In one embodiment, the bulk sortingsystem comprises at least 10 valves per meter. In other embodiments,more than 10 (such as 12, 15, 20, 100, 250, etc.,) valves per meter isemployed. More than one valve may be activated at once depending on thesize and/or weight of the object, as identified by the CNN.

The at least one mechanical kicker is arranged to kick the objects to adesired position. The desired position may be a different part of thebulk, a chute or container arranged in the direction of the at least onemechanical kicker or simply of the bulk sorting system. The kickers maybe electro-mechanical, pneumatic, spring-loaded and/or hydraulic,wherein the kickers are preferably pneumatic. In one embodiment, thebulk sorting system comprises at least 10 kickers per meter. In otherembodiments, more than 10 (such as 12, 15, 20, 100, 250 etc.,) kickersper meter is employed.

The objects are sorted using the analysis of the processing circuit.This may comprise directing the mechanical sorter to the objectsidentified by the CNN. This may further comprise sorting the objectsaccording to the set the object belongs to, the set being identified bythe CNN.

The mechanical sorter may be configured to separate the objects into atleast two streams. This allows the bulk sorting system to sort objectsin bulk into several streams, wherein each stream may e.g. comprise aspecific set of objects. As an example, bulk garbage may be sorted intoa stream of plastic, a stream of metal and a stream of other materials,wherein each stream is directed or conveyed to different places. Thesystem may be used for bulk sorting of food and mining materials in asimilar manner.

The mechanical sorter may be configured to eject unwanted objects fromthe bulk. This allows the bulk sorting system to remove unwanted objectsfrom the bulk. Ejected objects may be sorted into a specific containeror removed from the rest of the objects.

FIG. 4 shows an embodiment of a mechanical sorter that comprises anarray of valves 21 and one mechanical kicker 23. Several valves of thearray are arranged opposite a chute 25. The chute is intended forseparating a specific set of objects from the rest of the bulk. Whensuch an object is classified by the CNN, the processing circuit may beconfigured to instruct a number of the valves opposite the chute toactivate when the classified object passes by the valves, the numberbeing proportional to the size and/or weight of the object, asidentified by the CNN or using the output from the CNN.

The mechanical kicker of this mechanical sorter is arranged such that ifany unwanted object is too big and/or heavy to be moved by the array ofvalves, they may be ejected by the mechanical kicker before being movedbeyond the mechanical sorter.

The bulk sorting system may further comprise conveying the objects alongdetection range(s) of the at least one optical sensor to the mechanicalsorter using a conveyor belt 28. The objects may be conveyed in anynumber of ways, such as Brownian motion, fluid conveyance or air streamconveyance, however the inventors have found that the conveyor belt ismost efficient and does not interfere with the optical measurement orsorting. The conveyor belt may have a speed of approximately 2.5-3 m/s.

Other embodiments have the ejectors placed on the same side as the chuteis placed, alternatively sort in free fall or use conveyor belts formaterial transport between detection and ejection.

The mechanical sorter may be arranged to sort the objects as they travelalong the conveyor belt. The sorter may be arranged near the conveyorbelt and arranged to move the objects in a direction perpendicular tothe direction of motion of the conveyor belt.

The mechanical sorter may be arranged to sort the objects as they travelalong a chute or as they freefall. The chute or freefall may be arrangedat an end of a conveyor belt or along an edge of a conveyor belt. It ispreferential that the mechanical sorter is capable of reacting quicklyas the velocity of the objects are relatively hard to predict andcontrol in this embodiment. In one embodiment, the mechanical sorter hasan activation precision of less than 1 ms for small particles and lessthan 10 ms for bigger particles, such as bottles. The sorting may e.g.comprise moving the objects to a desired position such that wantedobjects chute or fall into a different container than unwanted objects.

The mechanical sorter may be arranged to sort the objects such thatdifferent overlapping and/or stacked objects, as analyzed by theprocessing circuit, are separated. As bulk sorting does not usuallycomprise any pre-sorting, overlapping and/or stacked objects arepossible and quite common. The processing circuit may identify theseobjects, preferably using the CNN. Once identified, the mechanicalsorter may be instructed or influenced by the processing circuit toseparate the objects. This may comprise e.g. opening an air valve for aprecise time and at a precise moment.

The mechanical sorter may be arranged to sort the objects such thatdifferent overlapping and/or stacked objects, as analyzed by theprocessing circuit, are treated as a single group of objects. As bulksorting does not usually comprise any pre-sorting or pre-processing toform a single stream of separated objects, overlapping and/or stackedobjects are possible and quite common. The processing circuit mayidentify these objects, preferably using the CNN.

Once identified, the mechanical sorter may not be capable or willing toseparate the objects, which may in turn depend on the mode of operationthat the processing circuit is in. As such, the different overlappingand/or stacked objects are treated as a single grouping that isprocessed as a single object. This processing may comprise e.g. ejectingthe group if it comprises any unwanted object or conserving the group ifit comprises a sufficient portion of wanted objects, measured usingsuitable measurements such as weight percentage or volume. Which actionis taken and whether wanted or unwanted objects are prioritized maydepend on settings or a use mode of the processing circuit/bulk sortingsystem. In other words, in one embodiment, overlapping and/or stackedobjects are treated as a single group of objects, wherein for a group ofobjects comprising at least a first object type and a second objecttype, the mechanical sorter is configured to sort the group of objectsas either the first object type or the second object type based on apreference in the bulk sorting system.

In one embodiment, the neural network is not handling overlappingobjects in a different way than separated objects. However the neuralnetwork is forced to learn to separate overlapping objects by givingnegative feedback in the training process if overlapping objects aremerged, only partly found or not all found. To handle this difficulttask a lot of sample images with overlapping objects and correspondinglabels (describing the contours of the touching and overlapping objects)may be provided during training of the CNN.

A process known as data augmentation is used to increase the samplesize, classically by rotations, small scale changes, color changes,crops and others applied to the whole image. To generate a lot moredifferent overlaps, a special augmentation technique is used for theconveyer belt setup. By labelling object instances in all the capturedtraining data with their class and surrounding contour, the images inthe training process may be used as they were captured and with thepreviously described standard augmentation in the following novel way:

Starting with images captured from the conveyer belt without objects,the labelled object instances are placed on this image and allowed tooverlap. Objects which are completely covered by others are removed fromthe labelled ground truth of the newly augmented image.

If an object is placed at least partly on another object, the new imageinformation in this area has to be generated according to the sensorproperties—e.g.: for NIR, VIS and RGB data the information of the objecton top overwrites the previously existing information, for X-ray datathe Beer-Lamber law is applied and for laser data a 3D height profile iscalculated for the two objects and the scatter as well intensity valueof the top object are used.

The mechanical sorter may be configured to work differently depending onthe analysis of the processing circuit. For example, a large andlight-weight object, as analyzed by the processing circuit, will onlyuse a portion of the possible ejectors as not all are needed for alight-weight object and thereby energy is conserved. In the embodimentwith an air valve array, only half of the air valves that cover theobject are used, thus both conserving energy and generating lessturbulence. In a reciprocal manner, the mechanical sorter may use alarger portion of possible ejectors than usual to increase the yield ofthe sorters and decrease the precision of the sorting, which may bebeneficial for heavy objects.

For mechanical sorters with variable strength, this may further beadjusted based on the analysis of the processing circuit. Accordingly,light-weight objects may use less strength than heavy-weight objects.

Further, the analysis of the processing circuit may be used to find thecentre of gravity or boundaries of the objects and affect the mechanicalsorter to target these areas. This may be beneficial to increase theefficiency of the sorter and may further be necessary for certain typesor sorters that require sophisticated targeting.

Further, if the position of objects is not stable, it may be beneficialto increase the number of ejectors activated and the time window forejection to compensate for unpredictable movement.

In FIG. 5 a , a bulk sorting system comprising a near-infrared (NIR)scanner 30 arranged to scan the objects and an image sensor 32 arrangedto capture image data of the objects is shown. The multi- orhyperspectral data that is input into the CNN comprises the scan andimage data. While several types of optical sensors are shown to be usedin tandem in FIG. 5 a and other figures, each optical sensor may operateindependently or together with any other optical sensor. The opticalsensors may further be arranged to measure the radiation of theradiation source or comprise a separate radiation source.

The NIR scan comprises spectroscopic data regarding the absorption ofthe objects of this wavelength range. Other wavelengths may be used as areplacement or in addition to the NIR range, such as visible light,ultra-violet (UV) light or X-rays. The NIR light may be provided by theradiation source or a separate source. The scan may be represented asmulti- or hyperspectral data by acquiring a separate spectrum for eachpixel or equivalent of the scanned area.

The image data comprises e.g. RGB pixel values of the reflected color ofthe objects. This may be used for traditional image processing by theprocessing circuit, such as object recognition or image segmentation.The objects may be illuminated by the radiation source to enable themeasurement of the image data. The image data may be represented asmulti- or hyperspectral data by acquiring a separate value of the amountof red, green and blue in each pixel or equivalent of the scanned area.

In FIG. 5 b , a bulk sorting system comprising a laser triangulator 34arranged to measure 3D-data of the objects and an electromagneticdetector 40 arranged to measure electromagnetic properties of theobjects is shown.

The multi- or hyperspectral data that is input into the CNN comprisesthe 3D-data and electromagnetic properties. Note that theelectromagnetic detector is not an optical sensor, however it may becombined with any optical sensor to acquire data from the objects to beinput as multi- or hyperspectral data into the CNN.

The 3D-data may be measured using laser height intensity scanning. Thiscomprises a laser directed towards the objects and a collector, such asa camera, measuring the intensity of the laser. The collector isarranged to measure the intensity from the surface the laser is directedtowards to 10-20 cm above the surface. This allows the collector todetect where the laser hits the objects, as this will impact itsintensity as the object disrupts the path of the laser.

The laser is preferably a line laser arranged across the width of thesurface that carries the objects, perpendicular to the direction ofmotion of the objects. As such, an entire object may be measured atonce. The laser may be the radiation source or provided separately.

The collector may comprise a band pass filter to filter out ambientlight, such that only the intensity of the laser wave lengths ismeasured.

The 3D-data comprises e.g. mapping of the height of the objects. The3D-data may be represented as multi- or hyperspectral data by acquiringa height value of each pixel or equivalent of the scanned area.

The electromagnetic properties may be measured using e.g.characteristics of reflection of applied electromagnetic waves or aconduction of an applied current. This may be used for e.g. detectingmetallic objects. Arrays of single coils are used for detection of metaland arrays of balanced coils for distinguishing of different metaltypes.

It may not be possible to directly allocate measured electromagneticproperties to specific pixels to the degree of accuracy as with opticalmeasurements. The allocation may therefore be estimated, possibly withthe use of optical data to differentiate between objects. Theelectromagnetic properties may be represented as multi- or hyperspectraldata by allocating measured electromagnetic properties to estimatedpixels or equivalent of the scanned area.

In the embodiment of FIG. 5 b , the electromagnetic detector uses multi-or hyperspectral data from the laser triangulator in order to allocatemeasured electromagnetic properties the multi- or hyperspectral datawith improved accuracy of the allocation. The communication of themulti- or hyperspectral data to the electromagnetic detector is showndirectly from the laser triangulator, however it may occur through theprocessing circuit.

The allocation of the measured electromagnetic properties to the multi-or hyperspectral data may alternatively occur separately in theprocessing circuit after some or all of the measured electromagneticproperties and multi- or hyperspectral data have been collected andbefore they are input into the CNN.

In FIG. 6 a , a bulk sorting system comprising a laser scanner 36 with arotating polygon mirror arranged to measure laser scatter and/oranti-scatter properties of the objects, a pulsed LED emitter 38 arrangedto measure light anti-scatter properties of the objects and an X-raycamera 42 arranged to measure X-ray transmission of the objects isshown. The multi- or hyperspectral data that is input into the CNNcomprises the laser scatter and/or anti-scatter properties, the lightanti-scatter properties and the X-ray transmission of the objects.

The different optical sensors are shown as a single unit in FIG. 6 a ;however, they may be any number of units.

The laser scatter and/or anti-scatter properties of the objectscomprises information of the reflectivity and absorption of the emittedlaser light. This may e.g. be used for object recognition. The laserscatter and/or anti-scatter properties of the objects may be representedas multi- or hyperspectral data by acquiring the laser scatter and/oranti-scatter properties of each pixel or equivalent of the scanned area.

The laser scatter and/or anti-scatter properties of the objects aremeasured using a detector arranged to measure the intensity of the laserlight reflected by the objects.

The laser scanner with a rotating polygon mirror emits a point laserinto the rotating polygon mirror that reflects the laser in differentdirections. This allows several objects and scattering characteristicsin different directions to be measured without any targeting. The lasermay have any wavelength and a combination of several wavelengths or mayalternate between different wavelengths. The laser may be considered asthe radiation source or be provided separately. The receiver might beconfigured to measure the reflected laser light, the scattered laserlight or both. Alternatively, the receiver may measure florescenceeffects in the objects caused by the laser.

The LED emitter may be configured to emit several different wavelengths,such as six different colors. The LED emitter is preferably pulsed, suchthat each color is emitted as a separate pulse. The LED emitter may beconsidered as the radiation source or be provided separately. LEDemitters are beneficial in that they require little maintenance andcalibration and generate relatively little heat.

The light anti-scatter properties of the objects are measured by adetector, such as a linescan detector, preferably an InGaAs linescandetector, that measures the intensity of reflected LED light. The lightanti-scatter properties for each emitted color and background withoutany LED emission may be represented as multi- or hyperspectral data byacquiring the light anti-scatter properties of each pixel or equivalentof the scanned area.

The X-ray transmission comprises the permittivity of the objects toX-ray radiation and is measured by the X-ray camera arranged opposite tothe X-ray emitter on the other side of the objects. The permittivity ofthe objects to X-ray radiation may indicate their atomic density and/ortheir thickness. The X-ray emitter may be considered as the radiationsource or be provided separately. The X-ray camera may have a single,dual or multiple energy measurement range. The X-ray transmission may berepresented as multi- or hyperspectral data by acquiring a separatespectrum of transmission for each pixel or equivalent of the scannedarea.

X-ray fluorescence allows the detection of existing elements inparticles. The material will be excited by low-energy X-ray radiationand element specific fluorescence will be released. With an energydispersive X-ray sensor, this fluorescence may be measured andrepresented as multi- or hyperspectral data. The result of thefluorescence is information about presence of elements and theirconcentration.

FIG. 6 b shows a bulk sorting system comprising a hyperspectral camera44 arranged to scan the objects, wherein the multi- or hyperspectraldata comprises the scan data. The hyperspectral camera is beneficial inthat it is an efficient way to gather a lot of data to be used by theCNN.

FIG. 7 shows a flowchart for operations of the processing circuitaccording to one embodiment. Different data is collected from thedifferent sensors (note that the sensors included in FIG. 7 are only byway of example) and rectified into multi- or hyperspectral data. Thismulti- or hyperspectral data is input into the CNN. The CNN outputsdetected and classified and/or semantically segmented multi- orhyperspectral data to be post-processed.

The data collected in the example embodiment of FIG. 7 is HSIcorresponding to a hyperspectral image captured by a hyperspectralcamera 44, RGB corresponding to a Red-Green-Blue image captured by animage sensor 32, Scatter corresponding to laser scatter and/oranti-scatter properties captured by a laser scanner 36, Heightcorresponding to 3D-data captured by a laser triangulator 34 and EMcorresponding to electromagnetic properties captured by anelectromagnetic detector 40.

The post-processing comprises converting the output into something themechanical sorter may more easily interpret. This may compriseconfiguration instructions for the mechanical sorter to be createdbefore sorting occurs of the objects corresponding to the measured data.

The post-processing may e.g. comprise interpreting the classification tofind whether the detected objects are wanted or unwanted and creatinginstructions for the mechanical sorter regarding how to deal with these.These instructions may comprise a control schedule regarding whichmechanical ejectors to activate when.

These post-processed instructions are then fed to the mechanical sorterthat sorts the objects, which may comprise ejecting unwanted objects.

FIG. 8 shows a flowchart for hybrid operations of the processing circuitcomprising both a CNN and a traditional pattern recognition algorithm.In this embodiment, the processing circuit is configured to input atleast a part of the multi- or hyperspectral data into a patternrecognition algorithm. The results of both the CNN and the patternrecognition algorithm are then used by the mechanical sorter to sort theobjects, which may comprise post-processing the results before feedingthem to the mechanical sorter or not.

The hybrid operations are beneficial in that different information maybe obtained by the different analysis methods. While only two methodsare disclosed in FIGS. 8-9 , any type and number of optical and/orelectromagnetic analyses are possible to combine in this manner.

In FIG. 8 , a spectral scan, an RGB image, laser scatter and/oranti-scatter properties and electromagnetic properties are separatelyinput to the traditional pattern recognition algorithm without collatingthe different data to a single multi- or hyperspectral data, howeverinputting multi- or hyperspectral data is also possible.

The data input into the different analysis methods may be the same ordifferent. For example, in FIG. 8 , 3D-data is input into the CNN andnot into the traditional pattern recognition algorithm.

FIG. 9 shows a flowchart for hybrid operations of the processing circuitusing two computing units, shown as dashed boxes. This embodiment usesdifferent computing units for different types of data analysis, suchthat the computing unit to the left uses a CNN and the computing unit tothe right uses traditional pattern recognition and post-processing.

This is beneficial in that each computing unit may be optimized for aspecific type of analysis, thus increasing efficiency of the analysisstep and reducing the total time for sorting.

This separation may be implemented in a number of ways, such as severalcomputing units implementing the CNN analysis, or the post-processingbeing implemented in a separate computing unit. There is no limit to theamount of computing units that may cooperate for the operation of theprocessing circuit, and hybrid operations are possible but not required.

FIG. 10 shows a flowchart for a method for sorting objects in bulk. Themethod 100 comprises several steps that are performed in order.

The radiating step 110 comprises radiating the objects using at leastone radiation source. The radiation source may be an LED, a light bulb,a laser, an X-ray emitter and/or any other suitable radiation source.The radiation source may be arranged as different units, as ambientillumination and/or to radiate in different directions.

The capturing step 120 comprises capturing reflected radiation of theobjects using at least one optical sensor. The optical sensor may be anear-infrared (NIR) scanner, an image sensor, a laser triangulator, alaser scanner, a pulsed LED emitter and/or an X-ray camera.

The acquiring step 130 comprises acquiring the reflected radiation asmulti- or hyperspectral data. This may comprise converting the capturedradiation into multi- or hyperspectral data. This step 130 may beperformed in conjunction of the capturing step 120 as the reflectedradiation is captured. The multispectral data may be a multi-dimensionalimage where each pixel or equivalent (such as super pixel or grouping ofpixels) comprises a few bands (order of magnitude 1-10) with widespectrums (order of magnitude 100-1000 nm). The hyperspectral data maybe a multi-dimensional image where each pixel or equivalent (such assuper pixel or grouping of pixels) comprises many bands (order ofmagnitude 4-10000) with narrow spectrums (order of magnitude 1-50 nm).

The analysis step 140 comprises analysing the reflected radiation of theobjects by inputting the multi- or hyperspectral data into aconvolutional neural network (CNN) with at least two convolutionallayers in order to either detect and classify the objects in the multi-or hyperspectral data and/or semantically segment the multi- orhyperspectral data.

This step 140 is preferably performed by a processing circuit that mayinclude a processor, such as a central processing unit (CPU),microcontroller, or microprocessor. The processor is configured toexecute program code stored in a memory, in order to carry out at leastone step of the method for sorting objects in bulk.

The sorting step 160 comprises sorting, by a mechanical sorter, theobjects using the results of the analysis step 140. The mechanicalsorter may be at least one valve, mechanical kicker, robot arm or anyother suitable device capable of mechanically moving objects.

The method 100 may further comprise a post-processing step 150 thatcomprises post-processing the result of the analysis step 140 to convertthe result into something more suitable for use in the sorting step.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseinventions pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the inventions are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Moreover, although the foregoing descriptions and the associateddrawings describe exemplary embodiments in the context of certainexemplary combinations of elements and/or functions, it should beappreciated that different combinations of elements and/or functions maybe provided by alternative embodiments without departing from the scopeof the appended claims. In this regard, for example, differentcombinations of elements and/or functions than those explicitlydescribed above are also contemplated as may be set forth in some of theappended claims. In cases where advantages, benefits or solutions toproblems are described herein, it should be appreciated that suchadvantages, benefits and/or solutions may be applicable to some exampleembodiments, but not necessarily all example embodiments. Thus, anyadvantages, benefits or solutions described herein should not be thoughtof as being critical, required or essential to all embodiments or tothat which is claimed herein. Although specific terms are employedherein, they are used in a generic and descriptive sense only and notfor purposes of limitation.

1. A bulk sorting system for sorting objects in bulk comprising: atleast one radiation source arranged to radiate the objects, at least oneoptical sensor arranged to capture reflected radiation of the objectsand acquire the reflected radiation as multi- or hyperspectral data,comprising a near-infrared (NIR) scanner arranged to scan the objects,wherein the multi- or hyperspectral data comprises the scan data; aprocessing circuit configured to analyze the reflected radiation of theobjects by inputting the multi- or hyperspectral data into aconvolutional neural network (CNN) with at least two convolutionallayers in order to detect and classify the objects in the multi- orhyperspectral data; and a mechanical sorter configured to sort theobjects according to their classification using the analysis of theprocessing circuit such that different overlapping and/or stackedobjects, as analyzed by the processing circuit, are treated as a singlegroup of objects.
 2. The bulk sorting system according to claim 1,wherein the at least one optical sensor comprises an image sensorarranged to capture image data of the objects, wherein the multi- orhyperspectral data comprises the image data.
 3. The bulk sorting systemaccording to claim 1, wherein the at least one optical sensor comprisesa multi- or hyperspectral camera arranged to scan the objects, whereinthe multi- or hyperspectral data comprises the scan.
 4. The bulk sortingsystem according to claim 1, wherein the at least one optical sensorcomprises a laser triangulator arranged to measure 3D-data of theobjects, wherein the multi- or hyperspectral data comprises the 3D-data,wherein the measurement of 3D-data may comprise laser height intensityscanning.
 5. The bulk sorting system according to claim 1, furthercomprising an electromagnetic detector arranged to measureelectromagnetic properties of the objects, wherein the processingcircuit is further configured to analyze the electromagnetic propertiesby inputting the measured electromagnetic properties into the CNN inorder to detect and classify the objects in the multi- or hyperspectraldata.
 6. The bulk sorting system according to claim 1, wherein the atleast one optical sensor comprises a laser scanner (36) with a rotatingpolygon mirror arranged to measure laser scatter and/or anti-scatterproperties of the objects, a pulsed LED emitter arranged to measurelight anti-scatter properties of the objects and/or an X-ray cameraarranged to measure X-ray transmission of the objects; wherein themulti- or hyperspectral data comprises the laser scatter and/oranti-scatter properties, the light anti-scatter properties and/or theX-ray transmission of the objects, respectively.
 7. The bulk sortingsystem according to claim 1, further comprising conveying the objectsalong detection range(s) of the at least one optical sensor to themechanical sorter using a conveyor belt.
 8. The bulk sorting systemaccording to claim 1, wherein the mechanical sorter is furtherconfigured to separate the objects into at least two streams and/or toeject unwanted objects from the bulk.
 9. The bulk sorting systemaccording to claim 1, wherein for a group of objects comprising at leasta first object type and a second object type, the mechanical sorter isconfigured to sort the group of objects as either the first object typeor the second object type based on a preference in the bulk sortingsystem.
 10. The bulk sorting system according to claim 1, wherein themechanical sorter is arranged to target the center of gravity orboundaries of the objects, as analyzed by the processing circuit. 11.The bulk sorting system according to claim 1, further comprisingpost-processing the detected and classified multi- or hyperspectral datato configure the mechanical sorter before sorting occurs.
 12. The bulksorting system according to claim 1, wherein the processing circuit isfurther configured to input at least a part of the multi- orhyperspectral data into a pattern recognition algorithm; and wherein theresults of both the CNN and the pattern recognition algorithm are usedby the mechanical sorter to sort the objects according to theirdetection and classification and/or segmentation as verified by thepattern recognition algorithm.
 13. A method for sorting objects in bulkcomprising steps of: radiating the objects using at least one radiationsource; capturing reflected radiation of the objects using at least oneoptical sensor, comprising capturing a scan using a multi- orhyperspectral scanner; acquiring the reflected radiation, comprising thescan, as multi- or hyperspectral data; analyzing the reflected radiationof the objects by inputting the multi- or hyperspectral data into aconvolutional neural network (CNN) with at least two convolutionallayers in order to detect and classify the objects in the multi- orhyperspectral data; and sorting, by a mechanical sorter, the objectsaccording to their classification using the results of the analysis stepsuch that different overlapping and/or stacked objects, as analyzed bythe processing circuit, are treated as a single group of objects.
 14. Amethod for sorting objects in bulk according to claim 13, whereincapturing reflected radiation of the objects using at least one opticalsensor comprises capturing a scan using a NIR scanner.